{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Here is the jupyter notebook for those who have signed up for the [competition](https://codalab.lisn.upsaclay.fr/competitions/8440?secret_key=51d5952f-d68d-47d9-baef-6032445dea01) in NTU.\n",
    "\n",
    "## Goal\n",
    "The competition's goal is to maximize the accumulative return rate of the test input. Based on the current price information, your agent should generate a portfolio weight which will give the hightest return rate defined below. \n",
    "\n",
    "## Dataset\n",
    "In this competition, you will be given 1 complete datasets as training dataset and 2 incomplete datasets as test datasets, where there is no feature `close`.\n",
    "\n",
    "In both complete and incomplete datasets, there are 15 assets.\n",
    "### Data Visualization\n",
    "Here is a glance of how the train data look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>feature_5</th>\n",
       "      <th>feature_6</th>\n",
       "      <th>feature_7</th>\n",
       "      <th>feature_8</th>\n",
       "      <th>feature_9</th>\n",
       "      <th>feature_10</th>\n",
       "      <th>date</th>\n",
       "      <th>close</th>\n",
       "      <th>tic</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.008320</td>\n",
       "      <td>0.002999</td>\n",
       "      <td>-0.010062</td>\n",
       "      <td>-0.143726</td>\n",
       "      <td>0.005374</td>\n",
       "      <td>-0.010348</td>\n",
       "      <td>-0.011415</td>\n",
       "      <td>0.002751</td>\n",
       "      <td>-0.061124</td>\n",
       "      <td>-0.072128</td>\n",
       "      <td>-0.126481</td>\n",
       "      <td>0</td>\n",
       "      <td>14.765714</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.005332</td>\n",
       "      <td>0.005803</td>\n",
       "      <td>-0.005646</td>\n",
       "      <td>-0.225326</td>\n",
       "      <td>-0.005459</td>\n",
       "      <td>0.007794</td>\n",
       "      <td>-0.008521</td>\n",
       "      <td>-0.008222</td>\n",
       "      <td>-0.017393</td>\n",
       "      <td>-0.031253</td>\n",
       "      <td>-0.042012</td>\n",
       "      <td>0</td>\n",
       "      <td>63.759998</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.004561</td>\n",
       "      <td>0.002902</td>\n",
       "      <td>-0.017413</td>\n",
       "      <td>-0.136140</td>\n",
       "      <td>-0.003100</td>\n",
       "      <td>0.004561</td>\n",
       "      <td>-0.004885</td>\n",
       "      <td>-0.017185</td>\n",
       "      <td>-0.029824</td>\n",
       "      <td>-0.045096</td>\n",
       "      <td>-0.052918</td>\n",
       "      <td>0</td>\n",
       "      <td>48.240002</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.003094</td>\n",
       "      <td>0.003632</td>\n",
       "      <td>-0.009956</td>\n",
       "      <td>-0.181742</td>\n",
       "      <td>0.001482</td>\n",
       "      <td>0.002922</td>\n",
       "      <td>-0.007421</td>\n",
       "      <td>-0.004776</td>\n",
       "      <td>-0.002812</td>\n",
       "      <td>-0.013054</td>\n",
       "      <td>-0.001687</td>\n",
       "      <td>0</td>\n",
       "      <td>74.330002</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.015182</td>\n",
       "      <td>0.006958</td>\n",
       "      <td>-0.015182</td>\n",
       "      <td>-0.252029</td>\n",
       "      <td>0.009257</td>\n",
       "      <td>0.010322</td>\n",
       "      <td>-0.004803</td>\n",
       "      <td>-0.010627</td>\n",
       "      <td>-0.001714</td>\n",
       "      <td>-0.005438</td>\n",
       "      <td>0.006133</td>\n",
       "      <td>0</td>\n",
       "      <td>94.849998</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  \\\n",
       "0  -0.008320   0.002999  -0.010062  -0.143726   0.005374  -0.010348   \n",
       "0   0.005332   0.005803  -0.005646  -0.225326  -0.005459   0.007794   \n",
       "0  -0.004561   0.002902  -0.017413  -0.136140  -0.003100   0.004561   \n",
       "0  -0.003094   0.003632  -0.009956  -0.181742   0.001482   0.002922   \n",
       "0  -0.015182   0.006958  -0.015182  -0.252029   0.009257   0.010322   \n",
       "\n",
       "   feature_6  feature_7  feature_8  feature_9  feature_10  date      close  \\\n",
       "0  -0.011415   0.002751  -0.061124  -0.072128   -0.126481     0  14.765714   \n",
       "0  -0.008521  -0.008222  -0.017393  -0.031253   -0.042012     0  63.759998   \n",
       "0  -0.004885  -0.017185  -0.029824  -0.045096   -0.052918     0  48.240002   \n",
       "0  -0.007421  -0.004776  -0.002812  -0.013054   -0.001687     0  74.330002   \n",
       "0  -0.004803  -0.010627  -0.001714  -0.005438    0.006133     0  94.849998   \n",
       "\n",
       "   tic  \n",
       "0    0  \n",
       "0    1  \n",
       "0    2  \n",
       "0    3  \n",
       "0    4  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "train=pd.read_csv(\"data/train.csv\",index_col=0)\n",
    "train.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the RL setting, we view all 15 stocks's feature information as state, which is the input of your model.\n",
    "\n",
    "Here is a glance of what the state in timestamp 0 looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>feature_5</th>\n",
       "      <th>feature_6</th>\n",
       "      <th>feature_7</th>\n",
       "      <th>feature_8</th>\n",
       "      <th>feature_9</th>\n",
       "      <th>feature_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.008320</td>\n",
       "      <td>0.002999</td>\n",
       "      <td>-0.010062</td>\n",
       "      <td>-0.143726</td>\n",
       "      <td>0.005374</td>\n",
       "      <td>-0.010348</td>\n",
       "      <td>-0.011415</td>\n",
       "      <td>0.002751</td>\n",
       "      <td>-0.061124</td>\n",
       "      <td>-0.072128</td>\n",
       "      <td>-0.126481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.005332</td>\n",
       "      <td>0.005803</td>\n",
       "      <td>-0.005646</td>\n",
       "      <td>-0.225326</td>\n",
       "      <td>-0.005459</td>\n",
       "      <td>0.007794</td>\n",
       "      <td>-0.008521</td>\n",
       "      <td>-0.008222</td>\n",
       "      <td>-0.017393</td>\n",
       "      <td>-0.031253</td>\n",
       "      <td>-0.042012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.004561</td>\n",
       "      <td>0.002902</td>\n",
       "      <td>-0.017413</td>\n",
       "      <td>-0.136140</td>\n",
       "      <td>-0.003100</td>\n",
       "      <td>0.004561</td>\n",
       "      <td>-0.004885</td>\n",
       "      <td>-0.017185</td>\n",
       "      <td>-0.029824</td>\n",
       "      <td>-0.045096</td>\n",
       "      <td>-0.052918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.003094</td>\n",
       "      <td>0.003632</td>\n",
       "      <td>-0.009956</td>\n",
       "      <td>-0.181742</td>\n",
       "      <td>0.001482</td>\n",
       "      <td>0.002922</td>\n",
       "      <td>-0.007421</td>\n",
       "      <td>-0.004776</td>\n",
       "      <td>-0.002812</td>\n",
       "      <td>-0.013054</td>\n",
       "      <td>-0.001687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.015182</td>\n",
       "      <td>0.006958</td>\n",
       "      <td>-0.015182</td>\n",
       "      <td>-0.252029</td>\n",
       "      <td>0.009257</td>\n",
       "      <td>0.010322</td>\n",
       "      <td>-0.004803</td>\n",
       "      <td>-0.010627</td>\n",
       "      <td>-0.001714</td>\n",
       "      <td>-0.005438</td>\n",
       "      <td>0.006133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.015490</td>\n",
       "      <td>0.020722</td>\n",
       "      <td>-0.012310</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.036759</td>\n",
       "      <td>-0.000532</td>\n",
       "      <td>-0.014879</td>\n",
       "      <td>-0.015927</td>\n",
       "      <td>-0.017220</td>\n",
       "      <td>-0.024889</td>\n",
       "      <td>-0.023492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.028963</td>\n",
       "      <td>0.000527</td>\n",
       "      <td>-0.033702</td>\n",
       "      <td>-0.266210</td>\n",
       "      <td>0.019324</td>\n",
       "      <td>0.005588</td>\n",
       "      <td>-0.001936</td>\n",
       "      <td>0.006550</td>\n",
       "      <td>0.013578</td>\n",
       "      <td>0.027855</td>\n",
       "      <td>0.050881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.003177</td>\n",
       "      <td>0.001724</td>\n",
       "      <td>-0.008895</td>\n",
       "      <td>-0.344117</td>\n",
       "      <td>-0.001722</td>\n",
       "      <td>0.003314</td>\n",
       "      <td>-0.016001</td>\n",
       "      <td>-0.032739</td>\n",
       "      <td>-0.049022</td>\n",
       "      <td>-0.064718</td>\n",
       "      <td>-0.077355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.016988</td>\n",
       "      <td>0.003604</td>\n",
       "      <td>-0.018790</td>\n",
       "      <td>-0.104656</td>\n",
       "      <td>0.014095</td>\n",
       "      <td>0.005675</td>\n",
       "      <td>-0.006954</td>\n",
       "      <td>-0.006971</td>\n",
       "      <td>-0.005922</td>\n",
       "      <td>-0.010628</td>\n",
       "      <td>-0.012547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.000105</td>\n",
       "      <td>0.004117</td>\n",
       "      <td>-0.011188</td>\n",
       "      <td>-0.151913</td>\n",
       "      <td>-0.006502</td>\n",
       "      <td>-0.000981</td>\n",
       "      <td>-0.015120</td>\n",
       "      <td>-0.020006</td>\n",
       "      <td>-0.021709</td>\n",
       "      <td>-0.028802</td>\n",
       "      <td>-0.025797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.015910</td>\n",
       "      <td>0.004211</td>\n",
       "      <td>-0.019186</td>\n",
       "      <td>-0.202751</td>\n",
       "      <td>0.014238</td>\n",
       "      <td>0.008977</td>\n",
       "      <td>0.001443</td>\n",
       "      <td>0.007767</td>\n",
       "      <td>0.012779</td>\n",
       "      <td>0.010444</td>\n",
       "      <td>0.013697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.000720</td>\n",
       "      <td>0.001981</td>\n",
       "      <td>-0.006843</td>\n",
       "      <td>-0.194121</td>\n",
       "      <td>-0.000900</td>\n",
       "      <td>-0.013952</td>\n",
       "      <td>-0.034563</td>\n",
       "      <td>-0.030250</td>\n",
       "      <td>-0.024226</td>\n",
       "      <td>-0.025329</td>\n",
       "      <td>-0.024293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000162</td>\n",
       "      <td>0.004258</td>\n",
       "      <td>-0.003234</td>\n",
       "      <td>-0.309785</td>\n",
       "      <td>-0.004079</td>\n",
       "      <td>-0.005678</td>\n",
       "      <td>-0.016651</td>\n",
       "      <td>-0.010289</td>\n",
       "      <td>-0.011314</td>\n",
       "      <td>-0.013083</td>\n",
       "      <td>-0.004030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.021505</td>\n",
       "      <td>0.000796</td>\n",
       "      <td>-0.023098</td>\n",
       "      <td>-0.260476</td>\n",
       "      <td>0.023227</td>\n",
       "      <td>-0.004684</td>\n",
       "      <td>-0.020664</td>\n",
       "      <td>-0.038132</td>\n",
       "      <td>-0.051560</td>\n",
       "      <td>-0.066108</td>\n",
       "      <td>-0.076438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.004123</td>\n",
       "      <td>0.006109</td>\n",
       "      <td>-0.005651</td>\n",
       "      <td>-0.250988</td>\n",
       "      <td>-0.006072</td>\n",
       "      <td>0.004932</td>\n",
       "      <td>-0.005748</td>\n",
       "      <td>-0.011223</td>\n",
       "      <td>-0.010806</td>\n",
       "      <td>-0.017515</td>\n",
       "      <td>-0.020816</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  \\\n",
       "0  -0.008320   0.002999  -0.010062  -0.143726   0.005374  -0.010348   \n",
       "0   0.005332   0.005803  -0.005646  -0.225326  -0.005459   0.007794   \n",
       "0  -0.004561   0.002902  -0.017413  -0.136140  -0.003100   0.004561   \n",
       "0  -0.003094   0.003632  -0.009956  -0.181742   0.001482   0.002922   \n",
       "0  -0.015182   0.006958  -0.015182  -0.252029   0.009257   0.010322   \n",
       "0   0.015490   0.020722  -0.012310   0.000000  -0.036759  -0.000532   \n",
       "0  -0.028963   0.000527  -0.033702  -0.266210   0.019324   0.005588   \n",
       "0  -0.003177   0.001724  -0.008895  -0.344117  -0.001722   0.003314   \n",
       "0  -0.016988   0.003604  -0.018790  -0.104656   0.014095   0.005675   \n",
       "0  -0.000105   0.004117  -0.011188  -0.151913  -0.006502  -0.000981   \n",
       "0  -0.015910   0.004211  -0.019186  -0.202751   0.014238   0.008977   \n",
       "0  -0.000720   0.001981  -0.006843  -0.194121  -0.000900  -0.013952   \n",
       "0   0.000162   0.004258  -0.003234  -0.309785  -0.004079  -0.005678   \n",
       "0  -0.021505   0.000796  -0.023098  -0.260476   0.023227  -0.004684   \n",
       "0   0.004123   0.006109  -0.005651  -0.250988  -0.006072   0.004932   \n",
       "\n",
       "   feature_6  feature_7  feature_8  feature_9  feature_10  \n",
       "0  -0.011415   0.002751  -0.061124  -0.072128   -0.126481  \n",
       "0  -0.008521  -0.008222  -0.017393  -0.031253   -0.042012  \n",
       "0  -0.004885  -0.017185  -0.029824  -0.045096   -0.052918  \n",
       "0  -0.007421  -0.004776  -0.002812  -0.013054   -0.001687  \n",
       "0  -0.004803  -0.010627  -0.001714  -0.005438    0.006133  \n",
       "0  -0.014879  -0.015927  -0.017220  -0.024889   -0.023492  \n",
       "0  -0.001936   0.006550   0.013578   0.027855    0.050881  \n",
       "0  -0.016001  -0.032739  -0.049022  -0.064718   -0.077355  \n",
       "0  -0.006954  -0.006971  -0.005922  -0.010628   -0.012547  \n",
       "0  -0.015120  -0.020006  -0.021709  -0.028802   -0.025797  \n",
       "0   0.001443   0.007767   0.012779   0.010444    0.013697  \n",
       "0  -0.034563  -0.030250  -0.024226  -0.025329   -0.024293  \n",
       "0  -0.016651  -0.010289  -0.011314  -0.013083   -0.004030  \n",
       "0  -0.020664  -0.038132  -0.051560  -0.066108   -0.076438  \n",
       "0  -0.005748  -0.011223  -0.010806  -0.017515   -0.020816  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "technical_indicator_list=[\"feature_0\",\"feature_1\",\"feature_2\",\"feature_3\",\"feature_4\",\"feature_5\",\"feature_6\",\"feature_7\",\"feature_8\",\"feature_9\",\"feature_10\"]\n",
    "train.loc[0][technical_indicator_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reward Calculation\n",
    "Based on the state in timestamp 0, we use random portfolio weights to demonstrate how the reward of the random action is calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0040574414586163005"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generate random action\n",
    "random_score=np.random.rand(15)\n",
    "random_weights=np.exp(random_score)/np.sum(np.exp(random_score))\n",
    "# calculate return_rate for every single ticker of day 0\n",
    "close_price_0=train.loc[0].close.values\n",
    "close_price_1=train.loc[1].close.values\n",
    "single_return_rate=close_price_1/close_price_0-1\n",
    "# calculate the return rate of the random action\n",
    "return_rate_action=np.sum(random_weights*single_return_rate)\n",
    "return_rate_action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide a [Trading Environment](https://github.com/TradeMaster-NTU/TradeMaster/blob/main/env/PM/portfolio_for_EIIE.py) to automate the process of state stepping and reward calculation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Implementation\n",
    "Here, we provide a baseline for the competition using one of the algorithms in TradeMaster. You can follow the rest jupyter notebook to create a borderline submission.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installation\n",
    "\n",
    "Here we provide the [video tutorial](https://www.youtube.com/watch?v=7rtqFT9I4uo&t=12s) for you to install this project to better participate in the competition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RL enviornment & baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Environment Set Up\n",
    "First, we need to add the project to our system path because the setting of jupter notebook is a little different from that of py document.\n",
    "Then, we need to load the document we need to import the module we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "from agent.EIIE.model import EIIE_con, EIIE_lstm, EIIE_rnn, EIIE_critirc\n",
    "import argparse\n",
    "from agent.EIIE.util import *\n",
    "from env.PM.portfolio_for_EIIE import Tradingenv\n",
    "from logging import raiseExceptions\n",
    "from stat import S_ENFMT\n",
    "import torch.nn as nn\n",
    "import pandas as pd\n",
    "import sys\n",
    "from agent.EIIE.trader import trader\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Config and Hyperparameters Adjustment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a part where you can adjust the default hyperparameters\n",
    "More specifically, besides the hyperparameters the DPG need, it also contains information like the path of the config file of the training and validing environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument(\"--random_seed\",\n",
    "                    type=int,\n",
    "                    default=12345,\n",
    "                    help=\"random seed number\")\n",
    "parser.add_argument(\n",
    "    \"--env_config_path\",\n",
    "    type=str,\n",
    "    default=\"config/input_config/env/portfolio/portfolio_for_EIIE/\",\n",
    "    help=\"the path for storing the downloaded data\")\n",
    "parser.add_argument(\n",
    "    \"--net_type\",\n",
    "    choices=[\"conv\", \"lstm\", \"rnn\"],\n",
    "    default=\"rnn\",\n",
    "    help=\"the name of the model\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--num_hidden_nodes\",\n",
    "    type=int,\n",
    "    default=32,\n",
    "    help=\"the number of hidden nodes in lstm or rnn\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--num_out_channel\",\n",
    "    type=int,\n",
    "    default=2,\n",
    "    help=\"the number of channel\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--gamma\",\n",
    "    type=float,\n",
    "    default=0.99,\n",
    "    help=\"the gamma for DPG\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--model_path\",\n",
    "    type=str,\n",
    "    default=\"result/EIIE/trained_model\",\n",
    "    help=\"the path for trained model\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--result_path\",\n",
    "    type=str,\n",
    "    default=\"result/EIIE/test_result\",\n",
    "    help=\"the path for test result\",\n",
    ")\n",
    "parser.add_argument(\n",
    "    \"--num_epoch\",\n",
    "    type=int,\n",
    "    default=1,\n",
    "    help=\"the number of epoch we train\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agent Building & Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = parser.parse_args(args=[])\n",
    "agent=trader(args)\n",
    "agent.train_with_valid()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing & Generate Submission Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_information_1=pd.read_csv(\"data/test_input_1.csv\",index_col=0)\n",
    "test_information_2=pd.read_csv(\"data/test_input_2.csv\",index_col=0)\n",
    "technical_indicator=[\"feature_0\",\"feature_1\",\"feature_2\",\"feature_3\",\"feature_4\",\"feature_5\",\"feature_6\",\"feature_7\",\"feature_8\",\"feature_9\",\"feature_10\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "action_list_1=[]\n",
    "for date in test_information_1.index.unique():\n",
    "    s=test_information_1[test_information_1.index==date][technical_indicator].values\n",
    "    shape=s.shape\n",
    "    s=s.reshape(shape[0],1,shape[1])\n",
    "    s=torch.from_numpy(s).float()\n",
    "    action=agent.net(s)\n",
    "    #here the origional action for the environment consider the cash, which is more pracitical in real world, but for \n",
    "    #the competition, we only need the last 15 weights for the assets, therefore we need to normalize the result as well\n",
    "    action=action.detach().float().numpy()\n",
    "    action=action[1:]/np.sum(action[1:])\n",
    "    action_list_1.append(action)\n",
    "action_list_1=np.array(action_list_1)\n",
    "action_list_1=action_list_1.astype(float)\n",
    "np.save(\"result/EIIE/action/action1.npy\",action_list_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "action_list_2=[]\n",
    "for date in test_information_2.index.unique():\n",
    "    s=test_information_2[test_information_2.index==date][technical_indicator].values\n",
    "    shape=s.shape\n",
    "    s=s.reshape(shape[0],1,shape[1])\n",
    "    s=torch.from_numpy(s).float()\n",
    "    action=agent.net(s)\n",
    "    #here the origional action for the environment consider the cash, which is more pracitical in real world, but for \n",
    "    #the competition, we only need the last 15 weights for the assets, therefore we need to normalize the result as well\n",
    "    action=action.detach().float().numpy()\n",
    "    action=action[1:]/np.sum(action[1:])\n",
    "    action_list_2.append(action)\n",
    "action_list_2=np.array(action_list_2)\n",
    "action_list_2=action_list_2.astype(float)\n",
    "np.save(\"result/EIIE/action/action2.npy\",action_list_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, Compress the 2 file as a zip file as submission file.\n",
    "Please notice that the zip file should only contain this 2 files name after `action1.npy` and `action2.npy` with no external folder. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next is core code of the agent.\n",
    "## RL Environment Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TradeMaster, you need to provide 3 RL environment configuration before building an agent. The TradeMaster agents will automatically follow the process using the training RL environment to train the agent, test it in the validation environment and pick the best model of training and pose it in the testing environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_env_instance=agent.train_env_instance\n",
    "valid_env_instance=agent.valid_env_instance\n",
    "test_env_instance=agent.train_env_instance\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reseting the environment, means to clear all the history and start from the begining. \n",
    "# It will return the initial state  \n",
    "# Here is an example of posing the random action to the train_env_instance\n",
    "s=train_env_instance.reset()\n",
    "action=np.random.rand(16)\n",
    "done=False\n",
    "while not done:\n",
    "    old_state = s\n",
    "    s, reward, done, _ =train_env_instance.step(action)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "next is the core code of EIIE, I will decompose it so that you can understand the process of training, making it easier for you to build your own agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the net\n",
    "# Just like the supervised leanrning process, we need a net to regress something, here the EIIE is Actor-Critic RL model where we need a actor to to generate policy and a critic\\\n",
    "# to judge whether the state is good or not\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "class EIIE_con(torch.nn.Module):\n",
    "    def __init__(self, in_channels, out_channels, length, kernel_size=3):\n",
    "        super(EIIE_con, self).__init__()\n",
    "        self.in_channels = in_channels\n",
    "        self.out_channels = out_channels\n",
    "        self.kernel_size = kernel_size\n",
    "        self.length = length\n",
    "        self.act = torch.nn.ReLU(inplace=False)\n",
    "        self.con1d = nn.Conv1d(self.in_channels,\n",
    "                               self.out_channels,\n",
    "                               kernel_size=3)\n",
    "        self.con2d = nn.Conv1d(self.out_channels,\n",
    "                               1,\n",
    "                               kernel_size=self.length - self.kernel_size + 1)\n",
    "        self.con3d = nn.Conv1d(1, 1, kernel_size=1)\n",
    "        self.para = torch.nn.Parameter(torch.ones(1).requires_grad_())\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.permute(0, 2, 1)\n",
    "        x = self.con1d(x)\n",
    "        x = self.act(x)\n",
    "        x = self.con2d(x)\n",
    "        x = self.act(x)\n",
    "        x = self.con3d(x)\n",
    "        x = x.view(-1)\n",
    "\n",
    "        # self.linear2 = nn.Linear(len(x), len(x) + 1)\n",
    "        # x = self.linear2(x)\n",
    "        x = torch.cat((x, self.para), dim=0)\n",
    "        x = torch.softmax(x, dim=0)\n",
    "\n",
    "        return x\n",
    "\n",
    "\n",
    "class EIIE_lstm(nn.Module):\n",
    "    def __init__(self, n_features, layer_num, n_hidden):\n",
    "        super(EIIE_lstm, self).__init__()\n",
    "        self.n_features = n_features\n",
    "        self.n_hidden = n_hidden\n",
    "        self.n_layers = layer_num\n",
    "        self.lstm = nn.LSTM(input_size=n_features,\n",
    "                            hidden_size=self.n_hidden,\n",
    "                            num_layers=self.n_layers,\n",
    "                            batch_first=True)\n",
    "        self.linear = nn.Linear(self.n_hidden, 1)\n",
    "        self.con3d = nn.Conv1d(1, 1, kernel_size=1)\n",
    "        self.para = torch.nn.Parameter(torch.ones(1).requires_grad_())\n",
    "\n",
    "    def forward(self, x):\n",
    "        lstm_out, _ = self.lstm(x)\n",
    "        x = self.linear(lstm_out[:, -1, :]).view(-1, 1, 1)\n",
    "        x = self.con3d(x)\n",
    "        x = x.view(-1)\n",
    "        x = torch.cat((x, self.para), dim=0)\n",
    "        x = torch.softmax(x, dim=0)\n",
    "        return x\n",
    "\n",
    "\n",
    "class EIIE_rnn(nn.Module):\n",
    "    def __init__(self, n_features, layer_num, n_hidden):\n",
    "        super(EIIE_rnn, self).__init__()\n",
    "        self.n_features = n_features\n",
    "        self.n_hidden = n_hidden\n",
    "        self.n_layers = layer_num\n",
    "        self.rnn = nn.RNN(input_size=n_features,\n",
    "                          hidden_size=self.n_hidden,\n",
    "                          num_layers=self.n_layers,\n",
    "                          batch_first=True)\n",
    "        self.linear = nn.Linear(self.n_hidden, 1)\n",
    "        self.con3d = nn.Conv1d(1, 1, kernel_size=1)\n",
    "        self.para = torch.nn.Parameter(torch.ones(1).requires_grad_())\n",
    "\n",
    "    def forward(self, x):\n",
    "        lstm_out, _ = self.rnn(x)\n",
    "        x = self.linear(lstm_out[:, -1, :]).view(-1, 1, 1)\n",
    "        x = self.con3d(x)\n",
    "        x = x.view(-1)\n",
    "        x = torch.cat((x, self.para), dim=0)\n",
    "        x = torch.softmax(x, dim=0)\n",
    "        return x\n",
    "\n",
    "\n",
    "class EIIE_critirc(nn.Module):\n",
    "    def __init__(self, n_features, layer_num, n_hidden):\n",
    "        super(EIIE_critirc, self).__init__()\n",
    "        self.n_features = n_features\n",
    "        self.n_hidden = n_hidden\n",
    "        self.n_layers = layer_num\n",
    "        self.lstm = nn.LSTM(input_size=n_features,\n",
    "                            hidden_size=self.n_hidden,\n",
    "                            num_layers=self.n_layers,\n",
    "                            batch_first=True)\n",
    "        self.linear = nn.Linear(self.n_hidden, 1)\n",
    "        self.con3d = nn.Conv1d(1, 1, kernel_size=1)\n",
    "        self.para = torch.nn.Parameter(torch.ones(1).requires_grad_())\n",
    "\n",
    "    def forward(self, x, a):\n",
    "        lstm_out, _ = self.lstm(x)\n",
    "        x = self.linear(lstm_out[:, -1, :]).view(-1, 1, 1)\n",
    "        x = self.con3d(x)\n",
    "        x = x.view(-1)\n",
    "        x = torch.cat((x, self.para, a), dim=0)\n",
    "        x = torch.nn.ReLU(inplace=False)(x)\n",
    "        number_nodes = len(x)\n",
    "        self.linear2 = nn.Linear(number_nodes, 1)\n",
    "        x = self.linear2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# here are the code for the trader, the key lies on the learn function\n",
    "class trader:\n",
    "    def __init__(self):\n",
    "        self.num_epoch = 10\n",
    "        self.GPU_IN_USE = torch.cuda.is_available()\n",
    "        self.device = torch.device('cpu' if self.GPU_IN_USE else 'cpu')\n",
    "        self.model_path = \"result/EIIE/trained_model\"\n",
    "        if not os.path.exists(self.model_path):\n",
    "            os.makedirs(self.model_path)\n",
    "        self.result_path = \"result/EIIE/test_result\"\n",
    "        if not os.path.exists(self.result_path):\n",
    "            os.makedirs(self.result_path)\n",
    "        self.train_env_instance = train_env_instance\n",
    "        self.valid_env_instance = valid_env_instance\n",
    "        self.test_env_instance = test_env_instance\n",
    "        self.day_length = 10\n",
    "        self.input_channel = 11\n",
    "        self.net = EIIE_con(self.input_channel, 2,\n",
    "                           self.day_length)\n",
    "        \n",
    "        self.critic = EIIE_critirc(self.input_channel, 1,\n",
    "                                  32)\n",
    "        self.test_action_memory = []  # to store the\n",
    "        self.optimizer_actor = torch.optim.Adam(self.net.parameters(), lr=1e-4)\n",
    "        self.optimizer_critic = torch.optim.Adam(self.critic.parameters(),\n",
    "                                                 lr=1e-4)\n",
    "        self.memory_counter = 0\n",
    "        self.memory_capacity = 1000\n",
    "        self.s_memory = []\n",
    "        self.a_memory = []\n",
    "        self.r_memory = []\n",
    "        self.sn_memory = []\n",
    "        self.policy_update_frequency = 500\n",
    "        self.critic_learn_time = 0\n",
    "        self.gamma = 0.99\n",
    "        self.mse_loss = nn.MSELoss()\n",
    "        self.net = self.net.to(self.device)\n",
    "        self.critic = self.critic.to(self.device)\n",
    "\n",
    "    def store_transition(\n",
    "        self,\n",
    "        s,\n",
    "        a,\n",
    "        r,\n",
    "        s_,\n",
    "    ):  # 定义记忆存储函数 (这里输入为一个transition)\n",
    "\n",
    "        self.memory_counter = self.memory_counter + 1\n",
    "        if self.memory_counter < self.memory_capacity:\n",
    "            self.s_memory.append(s)\n",
    "            self.a_memory.append(a)\n",
    "            self.r_memory.append(r)\n",
    "            self.sn_memory.append(s_)\n",
    "        else:\n",
    "            number = self.memory_counter % self.memory_capacity\n",
    "            self.s_memory[number - 1] = s\n",
    "            self.a_memory[number - 1] = a\n",
    "            self.r_memory[number - 1] = r\n",
    "            self.sn_memory[number - 1] = s_\n",
    "\n",
    "    def compute_single_action(self, state):\n",
    "        state = torch.from_numpy(state).float().to(self.device)\n",
    "        action = self.net(state)\n",
    "        action = action.detach().cpu().numpy()\n",
    "        return action\n",
    "\n",
    "    def learn(self):\n",
    "        # here is the core of the trader, it shows how the updates coming out\n",
    "        # we first need to have some stored the transcation(s,a,r,s_) \n",
    "        length = len(self.s_memory)\n",
    "        out1 = random.sample(range(length), int(length / 10))\n",
    "        # random sample\n",
    "        s_learn = []\n",
    "        a_learn = []\n",
    "        r_learn = []\n",
    "        sn_learn = []\n",
    "        for number in out1:\n",
    "            s_learn.append(self.s_memory[number])\n",
    "            a_learn.append(self.a_memory[number])\n",
    "            r_learn.append(self.r_memory[number])\n",
    "            sn_learn.append(self.sn_memory[number])\n",
    "        self.critic_learn_time = self.critic_learn_time + 1\n",
    "        # for the transcation we have stored, we need to update the actor and critic\n",
    "        # for the actor, we need to comput the action and use the critic to judge the action\n",
    "        # we need to update the actor so that for every action it choose, it can gain more scores from a critic than other action \n",
    "        # for the critic , we simply use the td_error to update it because it is MDP\n",
    "\n",
    "        for bs, ba, br, bs_ in zip(s_learn, a_learn, r_learn, sn_learn):\n",
    "            #update actor\n",
    "            a = self.net(bs)\n",
    "            q = self.critic(bs, a)\n",
    "            a_loss = -torch.mean(q)\n",
    "            self.optimizer_actor.zero_grad()\n",
    "            a_loss.backward(retain_graph=True)\n",
    "            self.optimizer_actor.step()\n",
    "            #update critic\n",
    "            a_ = self.net(bs_)\n",
    "            q_ = self.critic(bs_, a_.detach())\n",
    "            q_target = br + self.gamma * q_\n",
    "            q_eval = self.critic(bs, ba.detach())\n",
    "            # print(q_eval)\n",
    "            # print(q_target)\n",
    "            td_error = self.mse_loss(q_target.detach(), q_eval)\n",
    "            # print(td_error)\n",
    "            self.optimizer_critic.zero_grad()\n",
    "            td_error.backward()\n",
    "            self.optimizer_critic.step()\n",
    "\n",
    "    def train_with_valid(self):\n",
    "        rewards_list = []\n",
    "        for i in range(self.num_epoch):\n",
    "            j = 0\n",
    "            done = False\n",
    "            s = self.train_env_instance.reset()\n",
    "            while not done:\n",
    "\n",
    "                old_state = s\n",
    "                action = self.net(torch.from_numpy(s).float())\n",
    "                s, reward, done, _ = self.train_env_instance.step(\n",
    "                    action.detach().numpy())\n",
    "                self.store_transition(\n",
    "                    torch.from_numpy(old_state).float().to(self.device),\n",
    "                    action,\n",
    "                    torch.tensor(reward).float().to(self.device),\n",
    "                    torch.from_numpy(s).float().to(self.device))\n",
    "                j = j + 1\n",
    "                if j % 200 == 1:\n",
    "\n",
    "                    self.learn()\n",
    "            all_model_path = self.model_path + \"/all_model/\"\n",
    "            best_model_path = self.model_path + \"/best_model/\"\n",
    "            if not os.path.exists(all_model_path):\n",
    "                os.makedirs(all_model_path)\n",
    "            if not os.path.exists(best_model_path):\n",
    "                os.makedirs(best_model_path)\n",
    "            torch.save(self.net,\n",
    "                       all_model_path + \"actor_num_epoch_{}.pth\".format(i))\n",
    "            torch.save(self.critic,\n",
    "                       all_model_path + \"critic_num_epoch_{}.pth\".format(i))\n",
    "            s = self.valid_env_instance.reset()\n",
    "            done = False\n",
    "            rewards = 0\n",
    "            while not done:\n",
    "\n",
    "                old_state = s\n",
    "                action = self.net(torch.from_numpy(s).float())\n",
    "                s, reward, done, _ = self.valid_env_instance.step(\n",
    "                    action.detach().numpy())\n",
    "                rewards = rewards + reward\n",
    "            rewards_list.append(rewards)\n",
    "        index = rewards_list.index(np.max(rewards_list))\n",
    "        actor_model_path = all_model_path + \"actor_num_epoch_{}.pth\".format(\n",
    "            index)\n",
    "        critic_model_path = all_model_path + \"critic_num_epoch_{}.pth\".format(\n",
    "            index)\n",
    "        self.net = torch.load(actor_model_path)\n",
    "        self.critic = torch.load(critic_model_path)\n",
    "        torch.save(self.net, best_model_path + \"actor.pth\")\n",
    "        torch.save(self.critic, best_model_path + \"critic.pth\")\n",
    "\n",
    "    def test(self):\n",
    "        s = self.test_env_instance.reset()\n",
    "        done = False\n",
    "        while not done:\n",
    "            old_state = s\n",
    "            action = self.net(torch.from_numpy(s).float())\n",
    "            s, reward, done, _ = self.test_env_instance.step(\n",
    "                action.detach().numpy())\n",
    "        df_return = self.test_env_instance.save_portfolio_return_memory()\n",
    "        df_assets = self.test_env_instance.save_asset_memory()\n",
    "        assets = df_assets[\"total assets\"].values\n",
    "        daily_return = df_return.daily_return.values\n",
    "        df = pd.DataFrame()\n",
    "        df[\"daily_return\"] = daily_return\n",
    "        df[\"total assets\"] = assets\n",
    "        if not os.path.exists(self.result_path):\n",
    "            os.makedirs(self.result_path)\n",
    "        df.to_csv(self.result_path + \"/result.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent=trader()\n",
    "agent.train_with_valid()\n",
    "test_information_1=pd.read_csv(\"data/test_input_1.csv\",index_col=0)\n",
    "test_information_2=pd.read_csv(\"data/test_input_2.csv\",index_col=0)\n",
    "technical_indicator=[\"feature_0\",\"feature_1\",\"feature_2\",\"feature_3\",\"feature_4\",\"feature_5\",\"feature_6\",\"feature_7\",\"feature_8\",\"feature_9\",\"feature_10\"]\n",
    "action_list_1=[]\n",
    "for date in test_information_1.index.unique():\n",
    "    s=test_information_1[test_information_1.index==date][technical_indicator].values\n",
    "    shape=s.shape\n",
    "    s=s.reshape(shape[0],1,shape[1])\n",
    "    s=torch.from_numpy(s).float()\n",
    "    action=agent.net(s)\n",
    "    #here the origional action for the environment consider the cash, which is more pracitical in real world, but for \n",
    "    #the competition, we only need the last 15 weights for the assets, therefore we need to normalize the result as well\n",
    "    action=action.detach().float().numpy()\n",
    "    action=action[1:]/np.sum(action[1:])\n",
    "    action_list_1.append(action)\n",
    "action_list_1=np.array(action_list_1)\n",
    "action_list_1=action_list_1.astype(float)\n",
    "np.save(\"result/EIIE_c/action/action1.npy\",action_list_1)\n",
    "action_list_2=[]\n",
    "for date in test_information_2.index.unique():\n",
    "    s=test_information_2[test_information_2.index==date][technical_indicator].values\n",
    "    shape=s.shape\n",
    "    s=s.reshape(shape[0],1,shape[1])\n",
    "    s=torch.from_numpy(s).float()\n",
    "    action=agent.net(s)\n",
    "    #here the origional action for the environment consider the cash, which is more pracitical in real world, but for \n",
    "    #the competition, we only need the last 15 weights for the assets, therefore we need to normalize the result as well\n",
    "    action=action.detach().float().numpy()\n",
    "    action=action[1:]/np.sum(action[1:])\n",
    "    action_list_2.append(action)\n",
    "action_list_2=np.array(action_list_2)\n",
    "action_list_2=action_list_2.astype(float)\n",
    "np.save(\"result/EIIE_c/action/action2.npy\",action_list_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, Compress the 2 file as a zip file as submission file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.13 ('TradeMaster')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13 (default, Mar 29 2022, 02:18:16) \n[GCC 7.5.0]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "a1f97403911abd3f02553c8f2b0c54537fddc7efadd9f5d3e31784db6e40c347"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
